{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline \n",
    "%matplotlib nbagg\n",
    "import tensorflow as tf\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython import display\n",
    "from data_generator import get_batch, print_valid_characters\n",
    "from tensorflow.python.framework.ops import reset_default_graph\n",
    "\n",
    "import tf_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recurrent Neural Networks\n",
    "\n",
    "Recurrent neural networks are the natural type of neural network to use for sequential data i.e. time series analysis, translation, speech recognition, biological sequence analysis etc. Recurrent neural networks works by recursively applying the same operation at each time step of the data sequence and having layers that pass information from previous time step to the current. It can therefore naturally handle input of varying length. Recurrent networks can be used for several prediction tasks including: sequence-to-class, sequence tagging, and sequence-to-sequence predictions.\n",
    "\n",
    "In this exercise we'll implement a Encoder-Decoder RNN based on the GRU unit for a simple sequence to sequence translation task. This type of models have shown impressive performance in Neural Machine Translation and Image Caption generation. \n",
    "\n",
    "For more in depth background material on RNNs please see [Supervised Sequence Labelling with Recurrent\n",
    "Neural Networks](https://www.cs.toronto.edu/~graves/preprint.pdf) by Alex Graves\n",
    "\n",
    "We know that LSTMs and GRUs are difficult to understand. A very good non-mathematical introduction is [Chris Olahs blog](http://colah.github.io/posts/2015-08-Understanding-LSTMs/). (All the posts are nice and cover various topics within machine-learning)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Encoder-Decoder\n",
    "In the encoder-decoder structure one RNN (blue) encodes the input and a second RNN (red) calculates the target values. One essential step is to let the encoder and decoder communicate. In the simplest approach you use the last hidden state of the encoder to initialize the decoder. Other approaches lets the decoder attend to different parts of the encoded input at different timesteps in the decoding process. \n",
    "\n",
    "<img src=\"files/enc-dec.png\", width=400>\n",
    "\n",
    "In our implementation we use a RNN with gated recurrent units (GRU) as encoder. We then use the last hidden state of the encoder ($h^{enc}_T$) as input to the decoder which is also a GRU RNN. \n",
    "\n",
    "### RNNs in TensorFlow\n",
    "TensorFlow has implementations of LSTM and GRU units. Both implementations assume that the input from the tensor below has the shape **(batch_size, seq_len, num_features)**, unless you have `time\\_major=True`. In this excercise we will use the GRU unit since it only stores a single hidden value per neuron (LSTMs stores two) and is approximately twice as fast as the LSTM unit.\n",
    "\n",
    "As stated above we will implement a Encoder-Decoder model. The simplest way to do this is to encode the input sequence using the Encoder model. We will then use the last hidden state of the Encoder $h^{enc}_T$ as input to the decoder model which then uses this information (simply a fixed length vector of numbers) to produce the targets. There is (at least) two ways to input $h^{enc}_T$ into the decoder\n",
    "\n",
    "1. Repeatly use $h^{enc}_T$ as input to the Decoder at each decode time step, as well as the previously computed word\n",
    "2. Intialize the decoder using $h^{enc}_T$ and run the decoder without any inputs\n",
    "\n",
    "In this exercise we will follow the second approach because it's easier to implement. To do this need to create a tensorflow layer that takes $h^{enc}_T$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Data\n",
    "Since RNN models can be very slow to train on real large datasets we will generate some simpler training data for this exercise. The task for the RNN is simply to translate a string of letters spelling the numbers between 0-9 into the corresponding numbers i.e\n",
    "\n",
    "\"one two five\" --> \"125#\" (we use # as a special end-of-sequence character)\n",
    "\n",
    "To input the strings into the RNN model we translate the characters into a vector integers using a simple translation table (i.e. 'h'->16, 'o'-> 17 etc). The code below prints a few input/output pairs using the *get_batch* function which randomy produces the data.\n",
    "\n",
    "Do note; that as showed in the illustration above for input to the decoder the end-of-sequence tag is flipped, and used in the beginning instead of the end. This tag is known as start-of-sequence, but often the end-of-sequence tag is just reused for this purpose.\n",
    "\n",
    "In the data loader below you will see two targets, target input and target output. Where the input will be used to compute the translation and output used for the loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "batch_size = 3\n",
    "inputs, inputs_seqlen, targets_in, targets_out, targets_seqlen, targets_mask, \\\n",
    "text_inputs, text_targets_in, text_targets_out = \\\n",
    "    get_batch(batch_size=batch_size, max_digits=2, min_digits=1)\n",
    "\n",
    "print \"input types:\", inputs.dtype, inputs_seqlen.dtype, targets_in.dtype, targets_out.dtype, targets_seqlen.dtype\n",
    "print print_valid_characters()\n",
    "print \"Stop/start character = #\"\n",
    "\n",
    "for i in range(batch_size):\n",
    "    print \"\\nSAMPLE\",i\n",
    "    print \"TEXT INPUTS:\\t\\t\\t\", text_inputs[i]\n",
    "    print \"TEXT TARGETS INPUT:\\t\\t\", text_targets_in[i]\n",
    "    print \"TEXT TARGETS OUTPUT:\\t\\t\", text_targets_out[i]\n",
    "    print \"ENCODED INPUTS:\\t\\t\\t\", inputs[i]\n",
    "    print \"INPUTS SEQUENCE LENGTH:\\t\\t\", inputs_seqlen[i]\n",
    "    print \"ENCODED TARGETS INPUT:\\t\\t\", targets_in[i]\n",
    "    print \"ENCODED TARGETS OUTPUT:\\t\\t\", targets_out[i]\n",
    "    print \"TARGETS SEQUENCE LENGTH:\\t\", targets_seqlen[i]\n",
    "    print \"TARGETS MASK:\\t\\t\\t\", targets_mask[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder Decoder model setup\n",
    "Below is the TensorFlow model definition. We use an embedding layer to go from integer representation to vector representation of the input.\n",
    "\n",
    "Note that we have made use of a custom decoder wrapper which can be found in `rnn.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# resetting the graph\n",
    "reset_default_graph()\n",
    "\n",
    "# Setting up hyperparameters and general configs\n",
    "MAX_DIGITS = 5\n",
    "MIN_DIGITS = 5\n",
    "NUM_INPUTS = 27\n",
    "NUM_OUTPUTS = 11 #(0-9 + '#')\n",
    "\n",
    "BATCH_SIZE = 100\n",
    "# try various learning rates 1e-2 to 1e-5\n",
    "LEARNING_RATE = 0.005\n",
    "X_EMBEDDINGS = 8\n",
    "t_EMBEDDINGS = 8\n",
    "NUM_UNITS_ENC = 10\n",
    "NUM_UNITS_DEC = 10\n",
    "\n",
    "\n",
    "# Setting up placeholders, these are the tensors that we \"feed\" to our network\n",
    "Xs = tf.placeholder(tf.int32, shape=[None, None], name='X_input')\n",
    "ts_in = tf.placeholder(tf.int32, shape=[None, None], name='t_input_in')\n",
    "ts_out = tf.placeholder(tf.int32, shape=[None, None], name='t_input_out')\n",
    "X_len = tf.placeholder(tf.int32, shape=[None], name='X_len')\n",
    "t_len = tf.placeholder(tf.int32, shape=[None], name='X_len')\n",
    "t_mask = tf.placeholder(tf.float32, shape=[None, None], name='t_mask')\n",
    "\n",
    "# Building the model\n",
    "\n",
    "# first we build the embeddings to make our characters into dense, trainable vectors\n",
    "X_embeddings = tf.get_variable('X_embeddings', [NUM_INPUTS, X_EMBEDDINGS],\n",
    "                               initializer=tf.random_normal_initializer(stddev=0.1))\n",
    "t_embeddings = tf.get_variable('t_embeddings', [NUM_OUTPUTS, t_EMBEDDINGS],\n",
    "                               initializer=tf.random_normal_initializer(stddev=0.1))\n",
    "\n",
    "# setting up weights for computing the final output\n",
    "W_out = tf.get_variable('W_out', [NUM_UNITS_DEC, NUM_OUTPUTS])\n",
    "b_out = tf.get_variable('b_out', [NUM_OUTPUTS])\n",
    "\n",
    "X_embedded = tf.gather(X_embeddings, Xs, name='embed_X')\n",
    "t_embedded = tf.gather(t_embeddings, ts_in, name='embed_t')\n",
    "\n",
    "# forward encoding\n",
    "enc_cell = tf.nn.rnn_cell.GRUCell(NUM_UNITS_ENC)#python.ops.rnn_cell.GRUCell\n",
    "_, enc_state = tf.nn.dynamic_rnn(cell=enc_cell, inputs=X_embedded,\n",
    "                                 sequence_length=X_len, dtype=tf.float32)\n",
    "# use below incase TF's makes issues\n",
    "#enc_state, _ = tf_utils.encoder(X_embedded, X_len, 'encoder', NUM_UNITS_ENC)\n",
    "#\n",
    "#enc_state = tf.concat(1, [enc_state, enc_state])\n",
    "\n",
    "# decoding\n",
    "# note that we are using a wrapper for decoding here, this wrapper is hardcoded to only use GRU\n",
    "# check out tf_utils to see how you make your own decoder\n",
    "dec_out, valid_dec_out = tf_utils.decoder(enc_state, t_embedded, t_len, \n",
    "                                          NUM_UNITS_DEC, t_embeddings,\n",
    "                                          W_out, b_out)\n",
    "\n",
    "# reshaping to have [batch_size*seqlen, num_units]\n",
    "out_tensor = tf.reshape(dec_out, [-1, NUM_UNITS_DEC])\n",
    "valid_out_tensor = tf.reshape(valid_dec_out, [-1, NUM_UNITS_DEC])\n",
    "# computing output\n",
    "out_tensor = tf.matmul(out_tensor, W_out) + b_out\n",
    "valid_out_tensor = tf.matmul(valid_out_tensor, W_out) + b_out\n",
    "# reshaping back to sequence\n",
    "b_size = tf.shape(X_len)[0] # use a variable we know has batch_size in [0]\n",
    "seq_len = tf.shape(t_embedded)[1] # variable we know has sequence length in [1]\n",
    "num_out = tf.constant(NUM_OUTPUTS) # casting NUM_OUTPUTS to a tensor variable\n",
    "out_shape = tf.concat(0, [tf.expand_dims(b_size, 0),\n",
    "                          tf.expand_dims(seq_len, 0),\n",
    "                          tf.expand_dims(num_out, 0)])\n",
    "out_tensor = tf.reshape(out_tensor, out_shape)\n",
    "valid_out_tensor = tf.reshape(valid_out_tensor, out_shape)\n",
    "# handling shape loss\n",
    "#out_tensor.set_shape([None, None, NUM_OUTPUTS])\n",
    "y = out_tensor\n",
    "y_valid = valid_out_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print all the variable names and shapes\n",
    "for var in tf.all_variables():\n",
    "    s = var.name + \" \"*(40-len(var.name))\n",
    "    print s, var.value().get_shape()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining the cost function, gradient clipping and accuracy\n",
    "Because the targets are categorical we use the cross entropy error.\n",
    "As the data is sequential we use the sequence to sequence cross entropy supplied in `tf_utils.py`.\n",
    "We use the Adam optimizer but you can experiment with the different optimizers implemented in [TensorFlow](https://www.tensorflow.org/versions/r0.10/api_docs/python/train.html#optimizers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def loss_and_acc(preds):\n",
    "    # sequence_loss_tensor is a modification of TensorFlow's own sequence_to_sequence_loss\n",
    "    # TensorFlow's seq2seq loss works with a 2D list instead of a 3D tensors\n",
    "    loss = tf_utils.sequence_loss_tensor(preds, ts_out, t_mask, NUM_OUTPUTS) # notice that we use ts_out here!\n",
    "    # if you want regularization\n",
    "    #reg_scale = 0.00001\n",
    "    #regularize = tf.contrib.layers.l2_regularizer(reg_scale)\n",
    "    #params = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)\n",
    "    #reg_term = sum([regularize(param) for param in params])\n",
    "    #loss += reg_term\n",
    "    # calculate accuracy\n",
    "    argmax = tf.to_int32(tf.argmax(preds, 2))\n",
    "    correct = tf.to_float(tf.equal(argmax, ts_out)) * t_mask\n",
    "    accuracy = tf.reduce_sum(correct) / tf.reduce_sum(t_mask)\n",
    "    return loss, accuracy, argmax\n",
    "\n",
    "loss, accuracy, predictions = loss_and_acc(y)\n",
    "loss_valid, accuracy_valid, predictions_valid = loss_and_acc(y_valid)\n",
    "\n",
    "# use lobal step to keep track of our iterations\n",
    "global_step = tf.Variable(0, name='global_step', trainable=False)\n",
    "# pick optimizer, try momentum or adadelta\n",
    "optimizer = tf.train.AdamOptimizer(LEARNING_RATE)\n",
    "# extract gradients for each variable\n",
    "grads_and_vars = optimizer.compute_gradients(loss)\n",
    "# add below for clipping by norm\n",
    "#gradients, variables = zip(*grads_and_vars)  # unzip list of tuples\n",
    "#clipped_gradients, global_norm = (\n",
    "#    tf.clip_by_global_norm(gradients, self.clip_norm) )\n",
    "#grads_and_vars = zip(clipped_gradients, variables)\n",
    "# apply gradients and make trainable function\n",
    "train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print all the variable names and shapes\n",
    "# notice that we now have the optimizer Adam as well!\n",
    "for var in tf.all_variables():\n",
    "    s = var.name + \" \"*(40-len(var.name))\n",
    "    print s, var.value().get_shape()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# as always, test the forward pass and initialize the tf.Session!\n",
    "# here is some dummy data\n",
    "batch_size=3\n",
    "inputs, inputs_seqlen, targets_in, targets_out, targets_seqlen, targets_mask, \\\n",
    "text_inputs, text_targets_in, text_targets_out = \\\n",
    "    get_batch(batch_size=batch_size, max_digits=7, min_digits=2)\n",
    "\n",
    "for i in range(batch_size):\n",
    "    print \"\\nSAMPLE\",i\n",
    "    print \"TEXT INPUTS:\\t\\t\\t\", text_inputs[i]\n",
    "    print \"TEXT TARGETS INPUT:\\t\\t\", text_targets_in[i]\n",
    "\n",
    "# restricting memory usage, TensorFlow is greedy and will use all memory otherwise\n",
    "gpu_opts = tf.GPUOptions(per_process_gpu_memory_fraction=0.2)\n",
    "# initialize the Session\n",
    "sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_opts))\n",
    "# test train part\n",
    "sess.run(tf.initialize_all_variables())\n",
    "feed_dict = {Xs: inputs, X_len: inputs_seqlen, ts_in: targets_in,\n",
    "             ts_out: targets_out, t_len: targets_seqlen}\n",
    "fetches = [y]\n",
    "res = sess.run(fetches=fetches, feed_dict=feed_dict)\n",
    "print \"y\", res[0].shape\n",
    "\n",
    "# test validation part\n",
    "fetches = [y_valid]\n",
    "res = sess.run(fetches=fetches, feed_dict=feed_dict)\n",
    "print \"y_valid\", res[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Generate some validation data\n",
    "X_val, X_len_val, t_in_val, t_out_val, t_len_val, t_mask_val, \\\n",
    "text_inputs_val, text_targets_in_val, text_targets_out_val = \\\n",
    "    get_batch(batch_size=5000, max_digits=MAX_DIGITS,min_digits=MIN_DIGITS)\n",
    "print \"X_val\", X_val.shape\n",
    "print \"t_out_val\", t_out_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# setting up running parameters\n",
    "val_interval = 5000\n",
    "samples_to_process = 3e5\n",
    "samples_processed = 0\n",
    "samples_val = []\n",
    "costs, accs_val = [], []\n",
    "plt.figure()\n",
    "try:\n",
    "    while samples_processed < samples_to_process:\n",
    "        # load data\n",
    "        X_tr, X_len_tr, t_in_tr, t_out_tr, t_len_tr, t_mask_tr, \\\n",
    "        text_inputs_tr, text_targets_in_tr, text_targets_out_tr = \\\n",
    "            get_batch(batch_size=BATCH_SIZE,max_digits=MAX_DIGITS,min_digits=MIN_DIGITS)\n",
    "        # make fetches\n",
    "        fetches_tr = [train_op, loss, accuracy]\n",
    "        # set up feed dict\n",
    "        feed_dict_tr = {Xs: X_tr, X_len: X_len_tr, ts_in: t_in_tr,\n",
    "             ts_out: t_out_tr, t_len: t_len_tr, t_mask: t_mask_tr}\n",
    "        # run the model\n",
    "        res = tuple(sess.run(fetches=fetches_tr, feed_dict=feed_dict_tr))\n",
    "        _, batch_cost, batch_acc = res\n",
    "        costs += [batch_cost]\n",
    "        samples_processed += BATCH_SIZE\n",
    "        #if samples_processed % 1000 == 0: print batch_cost, batch_acc\n",
    "        #validation data\n",
    "        if samples_processed % val_interval == 0:\n",
    "            #print \"validating\"\n",
    "            fetches_val = [accuracy_valid, y_valid]\n",
    "            feed_dict_val = {Xs: X_val, X_len: X_len_val, ts_in: t_in_val,\n",
    "             ts_out: t_out_val, t_len: t_len_val, t_mask: t_mask_val}\n",
    "            res = tuple(sess.run(fetches=fetches_val, feed_dict=feed_dict_val))\n",
    "            acc_val, output_val = res\n",
    "            samples_val += [samples_processed]\n",
    "            accs_val += [acc_val]\n",
    "            plt.plot(samples_val, accs_val, 'g-')\n",
    "            plt.ylabel('Validation Accuracy', fontsize=15)\n",
    "            plt.xlabel('Processed samples', fontsize=15)\n",
    "            plt.title('', fontsize=20)\n",
    "            plt.grid('on')\n",
    "            plt.savefig(\"out.png\")\n",
    "            display.display(display.Image(filename=\"out.png\"))\n",
    "            display.clear_output(wait=True)\n",
    "except KeyboardInterrupt:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#plot of validation accuracy for each target position\n",
    "plt.figure(figsize=(7,7))\n",
    "plt.plot(np.mean(np.argmax(output_val,axis=2)==t_out_val,axis=0))\n",
    "plt.ylabel('Accuracy', fontsize=15)\n",
    "plt.xlabel('Target position', fontsize=15)\n",
    "#plt.title('', fontsize=20)\n",
    "plt.grid('on')\n",
    "plt.show()\n",
    "#why do the plot look like this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises:\n",
    "\n",
    "1. The model has two GRU networks. The ```GRUEncoder``` and the ```GRUDecoder```.\n",
    "A GRU is parameterized by a update gate `z`, reset gate `r` and the cell `c`.\n",
    "Under normal circumstances, such as in the TensorFlow GRUCell implementation, these gates have been stacked for faster computation, but in the custom decoder each weight and bias are as described in the original [article for GRU](https://arxiv.org/abs/1406.1078).\n",
    "Thus we have the following weights and bias; ```{decoder/W_z_x:0, decoder/W_z_h:0, b_updategate, decoder/b_z:0, decoder/W_r_x:0, decoder/W_r_h:0, decoder/b_r:0, decoder/W_c_x:0, decoder/W_c_h:0, decoder/b_h:0}```.\n",
    "Try to explain the shape of ```decoder/W_z_x:0``` and ```decoder/W_z_h:0```. Why are they different? You can find the equations for the gru at: [GRU](http://lasagne.readthedocs.io/en/latest/modules/layers/recurrent.html#lasagne.layers.GRULayer). \n",
    "\n",
    "2. The GRUunit is able to ignore the input and just copy the previous hidden state. In the begining of training this might be desireable behaviour because it helps the model learn long range dependencies. You can make the model ignore the input by modifying initial bias values. What bias would you modify and how would you modify it? Again you'll need to refer to the GRU equations:  [GRU](http://lasagne.readthedocs.io/en/latest/modules/layers/recurrent.html#lasagne.layers.GRULayer)\n",
    "Further, if you look into `tf_utils.py` and search for the `decoder(...)` function, you will see that the init for each weight and bias can be changed.\n",
    "\n",
    "3. Try setting MIN_DIGITS and MAX_DIGITS to 20\n",
    "\n",
    "4. What is the final validation performance? Why do you think it is not better? Comment on the accuracy for each position in of the output symbols?\n",
    "\n",
    "5. Why do you think the validation performance looks more \"jig-saw\" like compared to FFN and CNN models?\n",
    "\n",
    "6. In the example we stack a softmax layer on top of a Recurrent layer. In the code snippet below explain how we can do that?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reset_default_graph()\n",
    "\n",
    "bs_, seqlen_, numinputs_ = 16, 140, 40\n",
    "x_pl_ = tf.placeholder(tf.float32, [bs_, seqlen_, numinputs_])\n",
    "gru_cell_ = tf.nn.rnn_cell.GRUCell(10)\n",
    "l_gru_, gru_state_ = tf.nn.dynamic_rnn(gru_cell_, x_pl_, dtype=tf.float32)\n",
    "l_reshape_ = tf.reshape(l_gru_, [-1, 10])\n",
    "\n",
    "l_softmax_ = tf.contrib.layers.fully_connected(l_reshape_, 11, activation_fn=tf.nn.softmax)\n",
    "l_softmax_seq_ = tf.reshape(l_softmax_, [bs_, seqlen_, -1])\n",
    "\n",
    "print \"l_input_\", x_pl_.get_shape()\n",
    "print \"l_gru_\", l_gru_.get_shape()\n",
    "print \"l_reshape_\", l_reshape_.get_shape()\n",
    "print \"l_softmax_\", l_softmax_.get_shape()\n",
    "print \"l_softmax_seq_\", l_softmax_seq_.get_shape()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Optional: You are interested in doing sentiment analysis on tweets, i.e classification as positive or negative. You decide read over the twitter seqeuence and use the last hidden state to do the classification. How can you modify the small network above to only outa single classification for network? Hints: look at the gru\\_state\\_ or the [tf.slice](https://www.tensorflow.org/versions/r0.10/api_docs/python/array_ops.html#slice) in the API.\n",
    "\n",
    "\n",
    "7. Optional: Bidirectional Encoder, Bidirectional Encoders are usually implemented by running a forward model and  a backward model (a forward model on a reversed sequence) separately and the concatenating them before parsing them on to the next layer. To reverse the sequence try looking at [tf.reverse_sequence](https://www.tensorflow.org/versions/r0.10/api_docs/python/array_ops.html#reverse_sequence)\n",
    "\n",
    "```\n",
    "enc_cell = tf.nn.rnn_cell.GRUCell(NUM_UNITS_ENC)\n",
    "_, enc_state = tf.nn.dynamic_rnn(cell=enc_cell, inputs=X_embedded,\n",
    "                                 sequence_length=X_len, dtype=tf.float32, scope=\"rnn_forward\")\n",
    "\n",
    "X_embedded_backwards = tf.reverse_sequence(X_embedded, tf.to_int64(X_len), 1)\n",
    "enc_cell_backwards = tf.nn.rnn_cell.GRUCell(NUM_UNITS_ENC)\n",
    "_, enc_state_backwards = tf.nn.dynamic_rnn(cell=enc_cell_backwards, inputs=X_embedded_backwards,\n",
    "                                 sequence_length=X_len, dtype=tf.float32, scope=\"rnn_backward\")\n",
    "\n",
    "enc_state = tf.concat(1, [enc_state, enc_state_backwards])\n",
    "```\n",
    "\n",
    "Note: you will need to double the NUM_UNITS_DEC, as it currently does not support different sizes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention Decoder (LSTM)\n",
    "Soft attention for recurrent neural networks have recently attracted a lot of interest.\n",
    "These methods let the Decoder model selective focus on which part of the encoder sequence it will use for each decoded output symbol.\n",
    "This relieves the encoder from having to compress the input sequence into a fixed size vector representation passed on to the decoder.\n",
    "Secondly we can interrogate the decoder network about where it attends while producing the ouputs.\n",
    "below we'll implement an LSTM-decoder with selective attention and show that it significantly improves the performance of the toy translation task.\n",
    "\n",
    "The siminal attention paper is https://arxiv.org/pdf/1409.0473v7.pdf\n",
    "\n",
    "The principle of attention models is simple. \n",
    "\n",
    "1. Use the encoder to get the hidden represention $\\{h^1_e, ...h^n_e\\}$ for each position in the input sequence. \n",
    "2. for timestep $t$ in the decoder do for $m = 1...n$ : $a_m = f(h^m_e, h^d_t)$. Where f is a function returning a scalar value. \n",
    "3. You can then normalize the sequence of scalars $\\{a_1, ... a_n\\}$ to get probablities $\\{p_1, ... p_n\\}$.\n",
    "4. Weight each $h^e_t$ by its probablity $p_t$ and sum to get $h_{in}$.\n",
    "5. Use $h_{in}$ as an additional input to the decoder. $h_{in}$ is recalculated each time the decoder is updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# resetting the graph\n",
    "reset_default_graph()\n",
    "\n",
    "# Setting up hyperparameters and general configs\n",
    "MAX_DIGITS = 10\n",
    "MIN_DIGITS = 10\n",
    "NUM_INPUTS = 27\n",
    "NUM_OUTPUTS = 11 #(0-9 + '#')\n",
    "\n",
    "BATCH_SIZE = 100\n",
    "# try various learning rates 1e-2 to 1e-5\n",
    "LEARNING_RATE = 0.005\n",
    "X_EMBEDDINGS = 8\n",
    "t_EMBEDDINGS = 8\n",
    "NUM_UNITS_ENC = 10\n",
    "NUM_UNITS_DEC = 10\n",
    "NUM_UNITS_ATTN = 20\n",
    "\n",
    "\n",
    "# Setting up placeholders, these are the tensors that we \"feed\" to our network\n",
    "Xs = tf.placeholder(tf.int32, shape=[None, None], name='X_input')\n",
    "ts_in = tf.placeholder(tf.int32, shape=[None, None], name='t_input_in')\n",
    "ts_out = tf.placeholder(tf.int32, shape=[None, None], name='t_input_out')\n",
    "X_len = tf.placeholder(tf.int32, shape=[None], name='X_len')\n",
    "t_len = tf.placeholder(tf.int32, shape=[None], name='X_len')\n",
    "t_mask = tf.placeholder(tf.float32, shape=[None, None], name='t_mask')\n",
    "\n",
    "# Building the model\n",
    "\n",
    "# first we build the embeddings to make our characters into dense, trainable vectors\n",
    "X_embeddings = tf.get_variable('X_embeddings', [NUM_INPUTS, X_EMBEDDINGS],\n",
    "                               initializer=tf.random_normal_initializer(stddev=0.1))\n",
    "t_embeddings = tf.get_variable('t_embeddings', [NUM_OUTPUTS, t_EMBEDDINGS],\n",
    "                               initializer=tf.random_normal_initializer(stddev=0.1))\n",
    "\n",
    "# setting up weights for computing the final output\n",
    "W_out = tf.get_variable('W_out', [NUM_UNITS_DEC, NUM_OUTPUTS])\n",
    "b_out = tf.get_variable('b_out', [NUM_OUTPUTS])\n",
    "\n",
    "X_embedded = tf.gather(X_embeddings, Xs, name='embed_X')\n",
    "t_embedded = tf.gather(t_embeddings, ts_in, name='embed_t')\n",
    "\n",
    "# forward encoding\n",
    "enc_cell = tf.nn.rnn_cell.GRUCell(NUM_UNITS_ENC)#python.ops.rnn_cell.GRUCell\n",
    "enc_out, enc_state = tf.nn.dynamic_rnn(cell=enc_cell, inputs=X_embedded,\n",
    "                                 sequence_length=X_len, dtype=tf.float32)\n",
    "# use below in case TF's does not work as intended\n",
    "#enc_state, _ = tf_utils.encoder(X_embedded, X_len, 'encoder', NUM_UNITS_ENC)\n",
    "#\n",
    "#enc_state = tf.concat(1, [enc_state, enc_state])\n",
    "\n",
    "# decoding\n",
    "# note that we are using a wrapper for decoding here, this wrapper is hardcoded to only use GRU\n",
    "# check out tf_utils to see how you make your own decoder\n",
    "dec_out, dec_out_valid, alpha_valid = \\\n",
    "    tf_utils.attention_decoder(enc_out, X_len, enc_state, t_embedded, t_len,\n",
    "                               NUM_UNITS_DEC, NUM_UNITS_ATTN, t_embeddings,\n",
    "                               W_out, b_out)\n",
    "\n",
    "# reshaping to have [batch_size*seqlen, num_units]\n",
    "out_tensor = tf.reshape(dec_out, [-1, NUM_UNITS_DEC])\n",
    "out_tensor_valid = tf.reshape(dec_out_valid, [-1, NUM_UNITS_DEC])\n",
    "# computing output\n",
    "out_tensor = tf.matmul(out_tensor, W_out) + b_out\n",
    "out_tensor_valid = tf.matmul(out_tensor_valid, W_out) + b_out\n",
    "# reshaping back to sequence\n",
    "b_size = tf.shape(X_len)[0] # use a variable we know has batch_size in [0]\n",
    "seq_len = tf.shape(t_embedded)[1] # variable we know has sequence length in [1]\n",
    "num_out = tf.constant(NUM_OUTPUTS) # casting NUM_OUTPUTS to a tensor variable\n",
    "out_shape = tf.concat(0, [tf.expand_dims(b_size, 0),\n",
    "                          tf.expand_dims(seq_len, 0),\n",
    "                          tf.expand_dims(num_out, 0)])\n",
    "out_tensor = tf.reshape(out_tensor, out_shape)\n",
    "out_tensor_valid = tf.reshape(out_tensor_valid, out_shape)\n",
    "# handling shape loss\n",
    "#out_tensor.set_shape([None, None, NUM_OUTPUTS])\n",
    "y = out_tensor\n",
    "y_valid = out_tensor_valid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def loss_and_acc(preds):\n",
    "    # sequence_loss_tensor is a modification of TensorFlow's own sequence_to_sequence_loss\n",
    "    # TensorFlow's seq2seq loss works with a 2D list instead of a 3D tensors\n",
    "    loss = tf_utils.sequence_loss_tensor(preds, ts_out, t_mask, NUM_OUTPUTS) # notice that we use ts_out here!\n",
    "    # if you want regularization\n",
    "    reg_scale = 0.00001\n",
    "    regularize = tf.contrib.layers.l2_regularizer(reg_scale)\n",
    "    params = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)\n",
    "    reg_term = sum([regularize(param) for param in params])\n",
    "    loss += reg_term\n",
    "    # calculate accuracy\n",
    "    argmax = tf.to_int32(tf.argmax(preds, 2))\n",
    "    correct = tf.to_float(tf.equal(argmax, ts_out)) * t_mask\n",
    "    accuracy = tf.reduce_sum(correct) / tf.reduce_sum(t_mask)\n",
    "    return loss, accuracy, argmax\n",
    "\n",
    "loss, accuracy, predictions = loss_and_acc(y)\n",
    "loss_valid, accuracy_valid, predictions_valid = loss_and_acc(y_valid)\n",
    "\n",
    "# use lobal step to keep track of our iterations\n",
    "global_step = tf.Variable(0, name='global_step', trainable=False)\n",
    "# pick optimizer, try momentum or adadelta\n",
    "optimizer = tf.train.AdamOptimizer(LEARNING_RATE)\n",
    "# extract gradients for each variable\n",
    "grads_and_vars = optimizer.compute_gradients(loss)\n",
    "# add below for clipping by norm\n",
    "#gradients, variables = zip(*grads_and_vars)  # unzip list of tuples\n",
    "#clipped_gradients, global_norm = (\n",
    "#    tf.clip_by_global_norm(gradients, self.clip_norm) )\n",
    "#grads_and_vars = zip(clipped_gradients, variables)\n",
    "# apply gradients and make trainable function\n",
    "train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# as always, test the forward pass and start the tf.Session!\n",
    "# here is some dummy data\n",
    "batch_size = 3\n",
    "inputs, inputs_seqlen, targets_in, targets_out, targets_seqlen, targets_mask, \\\n",
    "text_inputs, text_targets_in, text_targets_out = \\\n",
    "    get_batch(batch_size=3, max_digits=7, min_digits=2)\n",
    "\n",
    "for i in range(batch_size):\n",
    "    print \"\\nSAMPLE\",i\n",
    "    print \"TEXT INPUTS:\\t\\t\\t\", text_inputs[i]\n",
    "    print \"TEXT TARGETS INPUT:\\t\\t\", text_targets_in[i]\n",
    "\n",
    "# restricting memory usage, TensorFlow is greedy and will use all memory otherwise\n",
    "gpu_opts = tf.GPUOptions(per_process_gpu_memory_fraction=0.2)\n",
    "# initialize the Session\n",
    "sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_opts))\n",
    "# test train part\n",
    "sess.run(tf.initialize_all_variables())\n",
    "feed_dict = {Xs: inputs, X_len: inputs_seqlen, ts_in: targets_in,\n",
    "             ts_out: targets_out, t_len: targets_seqlen}\n",
    "fetches = [y]\n",
    "res = sess.run(fetches=fetches, feed_dict=feed_dict)\n",
    "print \"y\", res[0].shape\n",
    "\n",
    "# test validation part\n",
    "fetches = [y_valid]\n",
    "res = sess.run(fetches=fetches, feed_dict=feed_dict)\n",
    "print \"y_valid\", res[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# print all the variable names and shapes\n",
    "# notice that W_z is now packed, such that it contains both W_z_h and W_x_h, this is for optimization\n",
    "# further, we now have W_s, b_s. This is so NUM_UNITS_ENC and NUM_UNITS_DEC does not have to share shape ..!\n",
    "for var in tf.all_variables():\n",
    "    s = var.name + \" \"*(40-len(var.name))\n",
    "    print s, var.value().get_shape()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Generate some validation data\n",
    "X_val, X_len_val, t_in_val, t_out_val, t_len_val, t_mask_val, \\\n",
    "text_inputs_val, text_targets_in_val, text_targets_out_val = \\\n",
    "    get_batch(batch_size=5000, max_digits=MAX_DIGITS,min_digits=MIN_DIGITS)\n",
    "print \"X_val\", X_val.shape\n",
    "print \"t_out_val\", t_out_val.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# NOTICE - THIS MIGHT TAKE UPTO 30 MINUTES ON CPU..!\n",
    "# setting up running parameters\n",
    "val_interval = 5000\n",
    "samples_to_process = 3e5\n",
    "samples_processed = 0\n",
    "samples_val = []\n",
    "costs, accs = [], []\n",
    "plt.figure()\n",
    "try:\n",
    "    while samples_processed < samples_to_process:\n",
    "        # load data\n",
    "        X_tr, X_len_tr, t_in_tr, t_out_tr, t_len_tr, t_mask_tr, \\\n",
    "        text_inputs_tr, text_targets_in_tr, text_targets_out_tr = \\\n",
    "            get_batch(batch_size=BATCH_SIZE,max_digits=MAX_DIGITS,min_digits=MIN_DIGITS)\n",
    "        # make fetches\n",
    "        fetches_tr = [train_op, loss, accuracy]\n",
    "        # set up feed dict\n",
    "        feed_dict_tr = {Xs: X_tr, X_len: X_len_tr, ts_in: t_in_tr,\n",
    "             ts_out: t_out_tr, t_len: t_len_tr, t_mask: t_mask_tr}\n",
    "        # run the model\n",
    "        res = tuple(sess.run(fetches=fetches_tr, feed_dict=feed_dict_tr))\n",
    "        _, batch_cost, batch_acc = res\n",
    "        costs += [batch_cost]\n",
    "        samples_processed += BATCH_SIZE\n",
    "        #if samples_processed % 1000 == 0: print batch_cost, batch_acc\n",
    "        #validation data\n",
    "        if samples_processed % val_interval == 0:\n",
    "            #print \"validating\"\n",
    "            fetches_val = [accuracy_valid, y_valid, alpha_valid]\n",
    "            feed_dict_val = {Xs: X_val, X_len: X_len_val, ts_in: t_in_val,\n",
    "             ts_out: t_out_val, t_len: t_len_val, t_mask: t_mask_val}\n",
    "            res = tuple(sess.run(fetches=fetches_val, feed_dict=feed_dict_val))\n",
    "            acc_val, output_val, alp_val = res\n",
    "            samples_val += [samples_processed]\n",
    "            accs += [acc_val]\n",
    "            plt.plot(samples_val, accs, 'b-')\n",
    "            plt.ylabel('Validation Accuracy', fontsize=15)\n",
    "            plt.xlabel('Processed samples', fontsize=15)\n",
    "            plt.title('', fontsize=20)\n",
    "            plt.grid('on')\n",
    "            plt.savefig(\"out_attention.png\")\n",
    "            display.display(display.Image(filename=\"out_attention.png\"))\n",
    "            display.clear_output(wait=True)\n",
    "# NOTICE - THIS MIGHT TAKE UPTO 30 MINUTES ON CPU..!\n",
    "except KeyboardInterrupt:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#plot of validation accuracy for each target position\n",
    "plt.figure(figsize=(7,7))\n",
    "plt.plot(np.mean(np.argmax(output_val,axis=2)==t_out_val,axis=0))\n",
    "plt.ylabel('Accuracy', fontsize=15)\n",
    "plt.xlabel('Target position', fontsize=15)\n",
    "#plt.title('', fontsize=20)\n",
    "plt.grid('on')\n",
    "plt.show()\n",
    "#why do the plot look like this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "### attention plot, try with different i = 1, 2, ..., 1000\n",
    "i = 42\n",
    "\n",
    "column_labels = map(str, list(t_out_val[i]))\n",
    "row_labels = map(str, (list(X_val[i])))\n",
    "data = alp_val[i]\n",
    "fig, ax = plt.subplots()\n",
    "heatmap = ax.pcolor(data, cmap=plt.cm.Blues)\n",
    "\n",
    "# put the major ticks at the middle of each cell\n",
    "ax.set_xticks(np.arange(data.shape[1])+0.5, minor=False)\n",
    "ax.set_yticks(np.arange(data.shape[0])+0.5, minor=False)\n",
    "\n",
    "# want a more natural, table-like display\n",
    "ax.invert_yaxis()\n",
    "ax.xaxis.tick_top()\n",
    "\n",
    "ax.set_xticklabels(row_labels, minor=False)\n",
    "ax.set_yticklabels(column_labels, minor=False)\n",
    "\n",
    "plt.ylabel('output', fontsize=15)\n",
    "plt.xlabel('Attention plot', fontsize=15)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Plot of average attention weight as a function of the sequence position for each of \n",
    "#the 21 targets in the output sequence i.e. each line is the mean postion of the \n",
    "#attention for each target position.\n",
    "\n",
    "np.mean(alp_val, axis=0).shape\n",
    "plt.figure()\n",
    "plt.plot(np.mean(alp_val, axis=0).T)\n",
    "plt.ylabel('alpha', fontsize=15)\n",
    "plt.xlabel('Input Sequence position', fontsize=15)\n",
    "plt.title('Alpha weights', fontsize=20)\n",
    "plt.legend(map(str,range(1,22)), bbox_to_anchor=(1.125,1.0), fontsize=10)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Assignments for the attention decoder\n",
    "1. Explain what the attention plot show.\n",
    "2. Explain what the alphaweights show.\n",
    "3. Why are the alpha curve for the first digit narrow and peaked while later digits have alpha curves that are wider and less peaked?\n",
    "4. Why is attention a good idea for this problem? Can you think of other problems where attention is a good choice?\n",
    "5. Try setting MIN_DIGITS and MAX_DIGITS to 20\n",
    "6. Enable gradient clipping (under the loss codeblock)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
